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Abstract Wind prediction errors are known to affect the 
performance of automated air traffic management tools that rely 
on aircraft trajectory predictions. In particular, automated 
separation assurance tools, planned as part of the NextGen 
concept of operations, must be designed to account and 
compensate for the impact of wind prediction errors and other 
system uncertainties. In this paper we describe a high fidelity 
batch simulation study designed to estimate the separation 
distance required to compensate for the effects of wind- 
prediction errors throughout increasing traffic density on an 
airborne separation assistance system. These experimental runs 
are part of the Safety Performance of Airborne Separation 
experiment suite that examines the safety implications of 
prediction errors and system uncertainties on airborne 
separation assurance systems. In this experiment, wind- 
prediction errors were varied between zero and forty knots while 
traffic density was increased several times current traffic levels. 
In order to accurately measure the full unmitigated impact of 
wind-prediction errors, no uncertainty buffers were added to the 
separation minima. The goal of the study was to measure the 
impact of wind-prediction errors in order to estimate the 
additional separation buffers necessary to preserve separation 
and to provide a baseline for future analyses. Buffer estimations 
from this study will be used and verified in upcoming safety 
evaluation experiments under similar simulation conditions. 
Results suggest that the strategic airborne separation functions 
exercised in this experiment can sustain wind prediction errors 
up to 40kts at current day air traffic density with no additional 
separation distance buffer and at eight times the current day with 
no more than a 60% increase in separation distance buffer. 
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I. Introduction 

Assessing the safety effects of prediction errors and 
uncertainty on automation- supported functions in the Next 
Generation Air Transportation System [1] concept of 
operations is of foremost importance, particularly safety critical 
functions such as separation that involve human decision- 
making, both ground-based and airborne. The automation of 
separation functions must be designed to account for, and 
mitigate the impact of, information uncertainty and varying 
human response [2]. Wind-prediction errors are known to 
affect the perfonnance of trajectory prediction tools and ground 


based conflict probes [3, 4]. While the accuracy of wind field 
forecasts has improved in recent years and has been proven to 
be satisfactory for most applications it is known that occasional 
large errors can occur with potentially unacceptable impact to 
safety critical applications. For that reason, it is important to 
conduct extensive laboratory experiments to understand such 
effects and to develop and validate the necessary mitigation 
strategies to maintain safety standards. 

This study is part of a series of experiments that comprises 
batch simulation studies investigating the safety impact of 
prediction errors and system uncertainties on Airborne 
Separation Assistance Systems (ASAS) applications. The 
scenarios used in these experiments consist of randomized 
routes in a generic high-density airspace in which all aircraft 
are constrained to the same flight level. Sustained average 
traffic density is varied from 1 1.2 to 21 .4 aircraft per 10K nmi I. 2 , 
approximating up to about 12.5 times today’s traffic density in 
a similarly sized en route sector. Two previous experiments 
utilizing the same simulation platform have been conducted so 
far. Results from the first baseline study indicate that at five 
times the typical traffic density of today’s National Airspace 
System (NAS) and under the assumptions of the study, 
airborne separation can be safely performed [5]. hi the second 
study, pilot actions required by the ASAS automation to 
resolve traffic conflicts were varied over a wide range of 
response times, varying from 5 to 240 seconds. Results indicate 
that the strategic ASAS functions exercised in the experiment 
can sustain pilot response delays of up to 90 seconds and more, 
depending on the traffic density [2], 

The current study focuses on wind-prediction errors which 
are known to have a detrimental effect on the ability of 
automated trajectory prediction tools, which in turn affects the 
performance of both cockpit and ground based decision support 
tools that rely on accurate predictions. In this experiment 
wind-prediction errors were varied from 0 to 40 kts while 
traffic density was increased from five to 21 aircraft per 
10K mm 2 . Aircraft separation was set to the standard five nmi 
for en route airspace and no mitigation technique is used to 
compensate for the position uncertainty of the aircraft. The goal 
of the experiment was to measure the magnitude of the 
separation violations to determine the appropriate separation 
distance required to preserve safe separation. The resulting 
buffer estimations will be evaluated as part of subsequent 
research activities. 



This paper is organized as follows: Section II presents a 
brief summary of previous work. Section El describes the 
simulation platform, and Sections IV through VII describes the 
experiment design and results. Finally, Section VIE presents 
conclusions and future research directions. 

II. Background 

Integrated air/ground operational concepts have been 
proposed in which some aircraft crews exercise separation 
functions aided by ASAS tools on the flight deck, while air 
traffic controllers exercise ground based separation control for 
non-ASAS-equipped aircraft and terminal operations [6] often 
supported by various decision support tools. These decision 
aids rely on broadcast data-linked information that includes 
aircraft velocity vectors and limited flight plan information 
through a surveillance capability such as the Automatic 
Dependent Surveillance - Broadcast (ADS-B) [7]. The 
automation is designed to detect conflicts between aircraft and 
generate conflict resolution routes and conflict-free 
maneuvering advisories [8, 9] by building predictions of 
aircraft trajectories. One of the major difficulties associated 
with aircraft trajectory predictions are the assumptions that 
must be made regarding environmental conditions, aircraft 
dynamics, communication reliability and human operator’s 
performance. These assumptions introduce different degrees of 
uncertainties that must be accounted for by any automated 
system used in safety critical applications. Until now, safety 
evaluation of Separation Assurance (SA) applications has, for 
the most part, been based on studies using simulation tools that 
seldom include models of system uncertainties [10,11,12] and 
often make many simplifying assumptions such as perfect 
navigation performance and absence of prediction errors or off- 
nominal conditions. 

The effects of uncertainty on the performance of an 
automated conflict detection and resolution system were the 
focus of a recent experiment that studied uncertainty factors 
associated mostly with ground based modeling technology such 
as climb and descent speed profile, aircraft weight, and flight 
plan route intent. A more recent study [13] used an eight nmi 
separation for conflict detection and a 10 nmi separation for 
conflict resolution, for aircraft in level flight, to mitigate for 
uncertainties. Results pointed at key modeling issues such as 
climb trajectory prediction uncertainty or aircraft intent. 

The Safety Performance of Airborne Separation 
(SPAS) simulation suite of studies attempts to characterize and 
quantify the safety performance of ASAS applications using a 
simulation platform that includes high fidelity models of 
aircraft dynamics, flight management system, data-link 
communications, and conflict detection and resolution 
functions as described in the next section. The baseline set of 
runs for the SPAS experiment, completed m the spring of 2007, 
included scenarios with no system uncertainties or prediction 
errors [5]. Results from that study indicated that within the 
experimental conditions and assumptions, safety was preserved 
with no losses of separation observed for traffic densities much 
higher than current levels. For the baseline SPAS test runs 
described in the aforementioned study, ADS-B reception was 
perfect (i.e., all messages received at all ranges), frill aircraft 
trajectory intent was shared, the pilot responded correctly to all 


traffic alerts with no delays, and wind-predictions were 
accurate. Results showed that at five times the typical traffic 
density of today’s NAS, utilizing only airborne intent based, 
strategic conflict detection and resolution logic with a 10- 
minute look-ahead time, safe separation of aircraft can be 
maintained under the assumptions and conditions of the test. 

A second SPAS study [2] investigated the effects of 
human operator inattentiveness when interacting with cockpit 
based automated systems used for separation assistance. These 
effects were observed by varying pilot delays and 
responsiveness within wide ranges. Loss of Separation (LOS) 
count and resolution to First Loss of Separation (FLOS) time 
were evaluated to assess the performance of the system under 
the experimental conditions. An in-depth analysis of the 
underlying causes for the observed behavior revealed great 
stability of the airborne strategic resolution capability under a 
large range of conditions. 

Wind-prediction errors are known to affect the 
performance of trajectory prediction tools and ground based air 
traffic management tools. While the accuracy of wind field 
forecasts has improved in recent years it is still the case that 
large errors can occur, potentially impacting the performance 
of safety critical applications. The improved quality and 
increased availability of the new Rapid Update Cycle (RUC-2) 
wind forecasts are were shown [14] to significantly benefit air 
traffic management applications. A later study [15] investigated 
sources of wind forecast error differences between Rapid 
Update Cycle, version 1 (RUC-1), and the newer RUC-2. The 
study confirmed the previous work by demonstrating measured 
improvement on wind forecast quality and availability. The 
study also quantified the percentage of large infrequent vector 
errors was reduced which was shown to be 3% overall and 7% 
during peak months. “Such peak error periods have a strong 
impact on air traffic management automation tools particularly 
if they persist along a predicted trajectory for 20 mm or more. 
A 15-kt mean error in the along-track wind component over a 
20 min trajectory prediction for an aircraft will result in a 5 nmi 
position error.” [15] Two flight tests were conducted to 
evaluate the effects of multiple sources of errors on the 
trajectory prediction accuracy of both ground-based and 
airborne automation systems. [3] The main source of error for 
the ground based systems was found to be the predicted winds 
aloft. In fact, the wind error at cruise approached 80 knots for 
several runs. Results from these tests revealed the occasional 
existence of “large” errors in the predicted wind field (greater 
than 20 knots) that span multiple sectors for periods greater 
than several hours at a time. A more recent study [4] 
investigated the User Request Evaluation Tool’s (URET) 
prediction sensitivity to weather forecast error by altering 
RUC-2 weather forecast adding 20 or 60 knots to the wind 
magnitude, 45 or 90 degrees to the wind direction. A 
comparative statistical analysis provided evidence that the 
forecast errors in wind magnitude and direction had significant 
effect on the longitudinal trajectory error and a modest impact 
on retracted false alerts relative to controlled baseline URET 
runs with no errors. 



III. The Simulation Platform 

The simulation runs described herein were conducted in the 
Air Traffic Operations Laboratory (ATOL) at NASA Langley 
Research Center utilizing a distributed simulation platform that 
includes a cluster of aircraft simulators interconnected through 
a High Level Architecture communication and simulation 
infrastructure known as the Airspace & Traffic Operations 
Simulation (ATOS) [16] depicted in Fig. 1 



Figure 1. Airspace and Traffic Operations Simulation Platform 


The ATOS batch platform is comprised of hundreds of real- 
time aircraft simulators equipped with high fidelity, six degree- 
of-freedom dynamic airplane model, a Flight Management 
System (FMS), Mode-S ADS-B data-lmk capability, and a 
prototype airborne conflict management system called the 
Autonomous Operations Planner (AOP). Conceptually, the 
ATOS batch simulation platform is a multi-agent system 
composed of multiple distributed autonomous software agents 
modeling AS AS equipped aircraft flown by a pilot model. A 
rule-based pilot model was developed to “fly” the simulators 
and perfonn the basic pilot actions required to interact with the 
airborne conflict management system. In the current study, the 
pilot model was configured to react to all conflict alerts by 
requesting a resolution trajectory from AOP [8] and accepting 
it. The pilot model was also configured with a mean delay time 
and standard deviation of 500 and 1.5 milliseconds, 
respectively. These were the same values used in the initial 
baseline SPAS experiment [5]. 

The AOP [8], a NASA-developed research ASAS prototype 
was built for the study of advanced distributed air-ground air 
traffic management concepts. The intent-based conflict 
detection (CD) function of AOP uses state and intent data 
received from other traffic aircraft over ADS-B in combination 
with ownship state data, auto-flight mode settings, and flight 
plan information to deterministically predict future losses of 
separation. AOP also has a second, independent, CD system 
that uses state-vector projections to detect flight crew blunders 
and prediction faults of the intent-based CD system and other 
short time horizon conflicts. Conflict alerting is modeled after 
the multi-alert-level approach recommended by RTCA [17]. 
For conflict resolution (CR), AOP contains both strategic and 
tactical capabilities. Tactical CR refers to open-loop vectors or 
altitude changes to solve conflicts with no predetermined 


reconnection to the original trajectory. Strategic CR refers to 
the single action of modifying the flight plan such that the 
conflict is solved and the aircraft reconnects to the previous 
trajectory. The strategic resolution logic attempts to find a 
route that both conforms to a Required Time of Arrival (RTA) 
and is conflict-free for 20 minutes. Nominal look-ahead time is 
10 minutes, but the strategic CR will attempt resolutions with 
as little as two minutes to loss of separation (LOS). If a 
resolution is found, it is guaranteed to preserve separation in 
the absence of prediction errors, even if only one aircraft 
executes it. If a strategic resolution cannot be found in time, the 
system would normally transition to a tactical resolution phase 
(typically at three minutes to LOS), which was not present in 
this experiment. The result of not having the tactical back-up 
system is that conflicts irresolvable by the strategic system of 
both aircraft will result in separation loss. AOP also includes 
functions for conflict prevention, including at-a-glance 
maneuver restriction symbology for the flight crew and support 
for tactical / strategic trajectory probing (also known as 
provisional CD). These functions were not required for this 
study and were therefore disabled. 

IV. Experiment Description 

A. Goals of the Sudy 

The goal of this experiment was to determine an 
appropriate separation distance that will preserve safe 
separation in the presence of wind-prediction errors over an 
increasing range of air traffic densities. 

To determine this value, it was necessary to calculate the 
magnitude of separation violations in a series of simulations 
with varying wind-prediction errors over an increasing airspace 
density. This was accomplished by measuring the distance at 
the closest pomt of approach (CPA) between all aircraft in the 
simulation. This distance will be used as a predictor of buffer 
size in future experiments. In the current study, wind-prediction 
errors were varied from 0 to 40 kts and no mitigation technique 
was used to compensate for the position uncertainty of the 
surrounding aircraft. This CPA measure will provide a 
conservative (larger than required) estimate of the separation 
buffers given that the conditions of the experiment do not 
include tactical conflict resolutions. 

B. Experiment Scenarios and Assupmtions 

The test region, representing a notional en route airspace 
sector is modeled as a circular area with a diameter of 160 nmi. 
The test region is surrounded by an initialization region, the 
outer boundary of which is the location where aircraft are 
initialized in the simulation. This initialization method 
provides each aircraft’s AOP with a frill 10-minute look-ahead 
time for detecting conflicts that occur within the test region. 
Aircraft are generated at random points on the outer circle 
initially with straight trajectories that traverse the test region at 
random angles and continue to a waypoint with a required time 
of arrival An RTA constraint is placed outside the test area to 
force strategic conflict resolutions to comply with it, as an 
additional element of complexity. All aircraft in the simulation 
are ASAS equipped and fly at the same altitude so as to 
constrain the scenarios to lateral conflict resolutions only and 




to achieve higher traffic densities. For this study, the auto-flight 
system remained coupled to the FMS for lateral navigation 
such that there was no tactical maneuvering. No altitude 
changes were permitted. There were no communication errors 
or ADS-B message degradation due to signal range or 
interference. More details on the scenario design can be found 
in the Preliminary SPAS experiment report [5]. ADS-B 
communications mcluded full mtent data. A priority rule (i.e., 
right of way) system was in effect that prevented undesirable 
synchronicity of resolution maneuvers by both aircraft involved 
in a conflict. Aircraft given priority for a given conflict had 
their alerts delayed 3 minutes while the other conflicting 
aircraft were alerted immediately. In Fig. 2, aircraft B has an 
initial trajectory that is in a conflict with aircraft A, just 
entering the initialization zone. Aircraft B resolves the conflict 
by modifying its fly path as shown in the diagram. 

The state-based CD and the tactical CR capability were 
disabled to allow the current study to focus on strategic conflict 
management. 



C. Experimental Conditions 

A two-factor experimental matrix was designed in which 
the average traffic density was assigned four levels and the 
wind-prediction error was assigned five levels as shown in 
Table I. The four levels of traffic density tested represent a 
range of 3.5X to 12. 5X the current traffic density levels for 
high altitude, en-route airspace sectors in the NAS. More 
details on the traffic density calculations and how they relate to 
current air traffic conditions can be found in [5]. The average 
traffic density values shown in Table I were measured as the 
instantaneous count of aircraft inside the test region, sampled 
every ten seconds and later normalized to a sector size of ten 
thousand nmi 2 . The predicted wind speed and direction as well 
as the truth wind direction were set to zero for all test runs 
while the truth wind speeds were varied within a range of 0 to 
40 kts. Since the aircraft expect no winds, the truth winds 
represent wind-prediction errors for all the test conditions. 

An exploratory set of runs was also done for a single 
density (10) at all wind conditions in which the truth winds 
were set to 0 kts and the predicted wind speed was varied from 


0 to 40 kts. These runs were conducted in order to test whether 
there is a significant difference in the ASAS algorithm 
performance if wind error results primarily from poor 
prediction in the presence of winds versus over-prediction of 
winds when there are no winds. 


TABLE I. Experiment Conditions 


Traffic 
Density 
(#aircraft 
per 10K 
nmi 2 ) 

Pred. 

Wind 

Dir. 

(Deg) 

Predicted Wind 
Speed (Kts) 

Truth 

Wind 

Direction 

(Deg) 

Truth Wind 
Speed 
(Kts) 

~5 

0 ° 

0 

0 ° 

0 

10 

20 

30 

40 

~10 

0 

10 

20 

30 

40 

~16 

0 

10 

20 

30 

40 

~21 

0 

10 

20 

30 

40 

~10 

0 ° 

0 | 10 | 20 | 30^0 

0 ° 

0 ° 


Since aircraft routes are randomly generated and their mitial 
headings are uniformly distributed between 0° and 360°, 
aircraft trajectories traverse the test region with uniformly 
distributed heading angles. As a result, the modeled winds 
impact the aircraft during the simulation from all directions 
imposing a uniform distribution of head/tail and cross winds. 


D. Experiments Results 

A total of 45 simulation runs over the 20 conditions 
described above were conducted. This included three 
replications (18 real-time simulation hours) for the ~5 and ~10 
density cases in order to mcrease the number of conflicts to 
nearly the same level as for the ~16 and ~21 density cases 
which were run 6 hours each. Each run starts with different 
random seeds to assure independent replications. A total of 270 
real-time simulation hours were performed, representing 5120 
flight hours and 16295 unique paired conflicts. Flight hours are 
computed as the accumulated flight time within the test area of 
all the aircraft in the simulation run. Since flight hours and 
number of conflicts depend on the traffic density of the 
scenario the lower density scenarios were replicated to expand 
the sample size. The number of conflicts was computed as the 
total number of conflicts that were detected by the two aircraft 
involved whenever the predicted LOS would occur partially or 
completely inside the test region. Losses of separation (LOS) 
and the distance at the closest point of approach were computed 
during post-processing of the time-correlated aircraft states. 
Fig. 3 shows the normalized number of LOS for each test 
condition. The normalized LOS count is obtained by dividing 
the total number of LOS occurrences by the total flight hours at 
each test condition. As can be observed from this plot, both 
Wind-prediction Error and Traffic Density impact this value. 
There also appears to be a flattening of results at the extreme 
conditions where nearly one LOS occurs per Flight Hour. This 
is an artifact of the 5 nmi of separation used in the experiment. 

A large number of the LOS that occurred at the highest 
levels of traffic density and wind prediction error was either the 
result of limitations in trajectory turn modeling or simulation 
specific artifacts derived from the scenario conditions. This 



Figure 3: Normalized LOS (Total LOS/Total Flight Hours) at each Test 
Condition 


was expected since the traffic complexity becomes more 
constraining on the conflict resolution algorithms as density 
increases and, hence, solutions are more sensitive to the 
influence of wind-prediction errors and trajectory modeling 
errors of actually flown trajectories. Many of these losses 
corresponded to very small intrusions, (less than 0.05 mni). The 
separation minimum for this experiment was set to the en route 
standard separation of 5.0 nmi. No additional buffers were used 
to compensate for turn modeling uncertainty (known to be 
approximately 0.1 nmi) since the goal of this study was to 
actually determine the requisite value of the separation buffers. 
Earlier SPAS studies [5] used a 5.1 nmi separation minimum, 
corresponding to an additional buffer of 0. 1 mni. 



For each wind condition, the mean and standard deviation 
of the wind-prediction error for each aircraft was computed as 
the average over its entire route (inside the test area) sampled 
10 times per second. The results verified that the wind- 
prediction errors experienced by the aircraft during the test 
were as intended. Also, both the cross and head (/tail) wind 


components were computed to observe their individual effect 
on resulting LOS. No correlation was observed between the 
cross and head (/tail) wind errors and the magnitude of the 
LOS. 

The number of conflicts per flight hour is shown in Fig. 4. 
It is clear that, although that data is nearly the same value for 
all wind-prediction errors, it also climbs nearly linearly with 
respect to air traffic density. This indicates that the number of 
conflicts per flight hour is nearly independent of wind- 
prediction error (over the range tested), while fully responsive, 
linearly, with respect to aircraft density. Therefore, even 
though the number of conflicts increases with density, as 
expected, it does not vary significantly with respect to wind- 
prediction error over the range of wind-predictions errors 
tested. Consequently, this metric would not provide a good 
basis for estimation of separation buffers for wind-prediction 
error compensation. 

A number of other metrics collected during the experiment 
seem to have a significant correlation with wind-prediction 
errors such as secondary conflicts and delayed conflict 
detections. These metrics will be reported in a later study as 
they mainly address the performance of the ASAS application 
and are not central to the objective of this paper. 

E. Closest Point of Approach and Losses of Separation 

The main metric of interest in this study is the distance at 
the CPA observed for the traffic densities and wind-prediction 
error conditions of the experiment. This metric represents the 
magnitude of the intrusions and will be used to estimate the 
required additional separation buffers necessary to preserve 
safe separation. Both metrics, the number of LOS observed, 
and the distance at the closest point of approach (CPA) for each 



Figure 5. Minimum CPA calculated for each Test Condition 


LOS was computed during post processing of time correlated 
aircraft states. The minimum of those minimum separation 
distances is the CPA metric used in the study; i.e. the CPA 
metric is minimum value of all CPAs of each aircraft pair in a 
simulation for each simulation condition. For replicated runs, 
this number was calculated as the minimum of the minimums 


over all replications. Fig. 5 shows the actual calculated 
minimum for each condition run. The data for the extreme 
density case do not exhibit a trend and are not consistent. This 
is an indicator that the complexity limits for the solution space 
for the strategic AOP resolution logic under these test 
conditions has been reached. Since only lateral conflict 
resolutions are used which are limited to only three types (path 
stretch, path offset, and path intercept) and forced to comply 
with an RTA, it is expected that further enhancements to the 
genetic conflict resolution algorithm [8], introduction of 
vertical resolutions and constraint relaxation may help 
overcome this limit. The CPA metric as computed in this 
experiment was purposefully designed to capture the worst- 
case observation. Since in this experiment only the intent 
based, strategic conflict detection and resolution was exercised, 
it is likely that when the tactical capabilities of AOP are used, 
some of the LOS would have been prevented or the magnitude 
of the intrusions reduced, hi addition, the strategic resolutions 
were RTA constrained throughout all the conditions, further 
increasing the complexity of the test. 

The AOP constraint relaxation capability might have 
prevented some of the LOS resulting from very constrained 
multi-aircraft conflicts. However, as shown in Figure 5, CPAs 
show a clear trend as a response function of both traffic density 
and wind-prediction error for density levels no higher than 
16.4. The data at the highest average traffic level mdicates that 
under the conditions of the experiment, the complexity of the 
traffic is confounding the wind-prediction error effects and 
cannot be used in calculating an estimate for separation buffers 
for wind-prediction error compensation. At that level of 
density-induced complexity, we observe losses of separations 
that are due to complex multi-aircraft conflicts where strategic 
resolutions eventually fail to converge. This is an issue to be 
studied in future experiments that will include tactical CD&R 
and constraint relaxation methods in order to increase the 
traffic density levels. However, Fig. 3 and 5 both indicate a 
clear trend proportional to the magnitude of the wind errors and 
the traffic density up through 16. 

A detailed analysis of the causes and underlying conditions 
behind the observed LOS, the type of conflicts and resolution 
strategies, false and missed alerts, as well as the analysis of the 
trajectory prediction function performance is beyond the scope 
of this paper. That analysis will be included on an AOP 
performance study in the near future. In this paper, the primary 
objective was to analyze the aggregate CPA response to try to 
infer an estimate for a reasonable separation buffer for wind- 
prediction error compensation. 


TABLE n. Design Space Ranges 


Name 


Range 

(units) 

Mean 

Std Dev 

T 

Factor 

5.00-22.00 

(count) 

8.17 

2.16 

W 

Factor 

0-40 (kts) 

30 

21.21 

CPA 

Response 

0.19-5.13 

(mni) 

3.31 

1.55 


Initial evaluation of measured data showed a strong 
negative correlation between traffic density and CPA (-0.717) 
and a moderate positive correlation between wind-prediction 
errors and CPA (0.453). 



Figure 6 is a plot of the CPA surface model and the actual 
minimum CPA data points (Fig. 5) calculated from the 
experiment. This figure shows the observed minimum CPAs as 
dots above and below the surface. Note the degradation of the 
fit at the extreme density. Tins was discussed in Section IV. D 
and is due to the inability of the AOP to handle the complexity 
of this condition within the current constraints imposed on the 
algorithm. This represents approximately 12 times today’s 
traffic density and is beyond the design requirements specified 
by NextGen of 2025. 

The equation used to model the CPA surface in Fig. 6 is 
given below: 


V. CPA Building a Predictive Model 

While the CPA observations shown in Fig. 5 seem to 
indicate a trend, it is important to understand how the two 
factors, traffic density (T) and wind-prediction error (W), affect 
the response variable (CPA) if we are going to use it to 
estimate a separation buffer size. A model was built using the 
experimental design tool Design Expert™ [18]. In this case, a 
quadratic Response Surface model was used to fit the data 
shown in Fig. 5. The ranges of the two factors, T and W as 
well as the response variable are shown in Table II. 


CPA = 3.90780 + 0.3409 T - 0.024044 W -1.15105E-003 TW 
- 0.019570 T 2 -3.38857E-004 W 2 (1) 


The results from an Analysis of Variance (ANOVA) 
performed on the model are shown in Table III. The 
significance of each term in the model equation is indicated by 
the values for "Prob > F”. The model was found to be 
significant with F-value=15.17 and p-value<0.0001. There is 
only a 0.01% chance that a "Model F-Value" this large could 


occur due to noise. F-values less than 0.0500 indicate model 
terms that are significant. In this case T, W, T 2 are significant 
model terms. F-values greater than 0.1000 indicate the model 
terms are not as significant. In this case TW and W 2 were 
found not to be significant. In other words, changes to these 
two factors would not result in large effects on the response 
value. The relative significance of the model terms is reflected 
in the magnitudes of the coefficients in equation (1). 

TABLE m. Results from Design Expert™ ANOVA of the CPA 

MODEL 


Source 

F- Value 

Prob > F 

Model 

15.17 

<0.0001 

T 

17.73 

0.0009 

w 

5.03 

0.0416 

TW 

0.32 

0.5787 

T 2 

10.03 

0.0069 

w 2 

0.12 

0.7346 


Results of further analysis of the significance of the model 
coefficients are shown in Table IV. The relative impact of the 
model coefficients can be directly compared in this table since 
they are expressed in normalized units. The standard error is 
the estimated standard deviation of the coefficient estimate. 
The values for lower and upper 95% confidence intervals are 
shown in the last two columns of Table IV. If the range of the 
confidence interval includes zero then the coefficient is not 
significantly different from zero and may not be having a 
statistically significant effect on the response. This occurs for 
both the TW and W 2 terms, indicating they may not be 
significant. At the very least, these confidence values indicate 
the coefficients for these terms may be poorly estimated. 


TABLE IV. Results from Design Expert™ Model Coefficient 
Analysis 


Factor 

Coefficient 

Estimate 

Standard 

Error 

95% 

Cl 

Low 

95% 

Cl 

High 

Intercept 

-2.77 

1.17 

-5.27 

-0.26 

T 

-17.06 

4.05 

-25.76 

-8.37 

w 

-1.39 

0.62 

-2.71 

-0.060 

TW 

-0.52 

0.91 

-2.47 

1.44 

T 2 

-9.91 

3.13 

-16.62 

-3.20 

W 2 

-0.14 

0.39 

-0.98 

0.70 


This model can be used as a predictive tool to estimate the 
size of the separation buffers. It is also a valuable tool to 
explore the design space to begin to understand the valid ranges 
of the model factors, beyond which the solution is not valid. 
For example, the model indicates that the main effect of the 
density factor at its maximum level (21) is not a good predictor 
of performance. There are at least two observations that can be 
made regarding this effect. First, the highest tested density 
introduces a level of complexity to the scenario that confounds 
the winds effect. In other words, we cannot infer from the CPA 
value whether the LOS were the result of aircraft position 
uncertainty due to wind-prediction errors or too many multi- 
aircraft conflicts that were too complex to solve strategically. 


Second, the CPA minimum value depends on the separation m 
being used (i.e. 5 mni). Hence, response values too close to 
zero are not very reliable. 

VI. Analysis of Results 

This section presents an analysis of the results based on the 
predictive model described in the previous section, and 
develops an estimate for a reasonable separation buffer for 
wind-prediction error compensation in AS AS. 

The graph in Fig. 7 shows the surface model intersected by 
planes at boundaries of the design space for which the indicated 
separation buffers are shown by the contour projection on the 
horizontal plane. For instance, the bottom plane at Minimum 
CPA = 2 (intrusion of 3 nmi) indicates that a three mni buffer 
(5 nmi separation - 2 nmi CPA) is sufficient to prevent LOS 
for the factor ranges within the contour labeled “CPA 3 nmi 
buffer threshold.” hi other words, the lowest plane shows the 
boundary of the “qualifying data” e.g. the maximum ranges of 
the T and W factors for which the model appears reliable. 



Another view of the estimated separation buffers is 
provided by the transformed CPA response as shown in (2). 
The estimated buffer, B, is computed as the ceiling of the 
separation minimum (M) minus the computed CPA. This 
equation computes the minimum, integer buffer size required to 
compensate for the minimum measured CPA. In this 
experiment, the value of the separation minimum M is 5 nmi, 
hence the transformed response was computed by the 
expression in (3). 

B = f( M - CPA)1 (2) 


B = T(5 — CPA)1 (3) 

The contour graph in Fig. 8 is the horizontal projection of 
the buffer response surface shown in Fig. 7, in which the 
design points (Fig. 5) are shown as dots. The color intensity of 
both the contours and the data points indicate the value of 


minimum CPA. The horizontal axis displays the traffic density 
T over a range of five to 16 aircraft per 10K nmi 2 , excluding 
the highest level of the factor. The vertical axis displays the 
wind-prediction error W over a range of zero to 60 kts, 
extending the design space beyond the measured range. The 
contour lines indicate estimated buffer sizes in nmi (identified 
by the numbered squares) corresponding to the different levels 
of the factors T and W. The use of predictive models outside 
the design space is acceptable in this case since these are 
exploratory results to be used in upcoming validation studies. 



Figure 8: Contour graph of the buffer size model 


VII. Comparison of Varying Truth vs Predicticted 
Winds in Assessing Wind-Prediction Error 

A set of runs was conducted at a single average traffic 
density of approximately 10 aircraft per 10K nmi 2 in which the 
predicted winds were varied while the truth winds were held 
constant. 



Figure 9. Comparison of minimum CPA resulting from varying predicted 
winds vs. truth winds 


In these runs the truth winds were set to zero kts and the 
predicted wind speed was varied from 10 to 40 kts. These runs 
were conducted in order to determine whether there is a 
significant difference in the ASAS algorithm performance if 
wind error results primarily from poor prediction in the 
presence of winds versus over-prediction of winds when there 
are no winds. 

Fig. 9 shows the minimum CPA results from these runs. 
Based on this one sample of runs, it appears that results are 
different. More research needs to be conducted to determine 
both the reasons for and the significance of these differences. 
However, based on these results, the current buffer estimates 
appear to be adequate in that they work for both cases. 

VIII. Conclusions and Future work 

The primary goal of this experiment was to estimate the 
separation buffer size required to compensate for wind 
prediction error in an Airborne Separation Assistance System. 
Results suggest that the strategic airborne separation functions 
exercised in this experiment can sustain wind prediction errors 
up to 40kts at current day air traffic density with no additional 
separation distance buffer and at eight tunes the current day 
with no more than a 60% increase in separation distance buffer. 

An experimental model for separation buffer estimation 
was developed to characterize the data. It was shown that this 
model could be used to determine an estimate to meet the goals 
of this study. The constructed model can also be used to 
continue the exploration of safety performance of automated 
separation tools in the presence of wind-prediction errors. A 
methodology in building a predictive model was developed that 
can be used to explore the effects of other sources of error and 
system degradation and human performance in order to 
mitigate system uncertainties m automated separation tools. 
This research will continue with the testing and validation of 
the estimated buffers reported in this paper in upcoming safety 
evaluation experiments under similar simulation conditions. 
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